On the Expressive Flexibility of Self-Attention Matrices

Transformer networks are able to capture patterns in data coming from many domains (text, images, videos, proteins, etc.) with little or no change architecture components. We perform a theoretical analysis of the core component responsible for signal propagation between elements, i.e. self-attention matrix. ask following question: Can matrix approximate arbitrary patterns? How small is query dimension d required such approximation? Our first result shows that task deciding whether approximation given pattern possible not NP-hard fixed greater than one. In practice, typically exhibits two properties: it sparse, and changes dynamically depending on input module. Motivated by this observation, we show can provably sparse matrices. While parameters fixed, various matrices be approximated only modifying inputs. proof based random projection technique uses seminal Johnson-Lindenstrauss lemma. particular, that, order any up precision defined terms preserving element ratios, grows logarithmically sequence length n.